Image processing apparatus and image processing method

ABSTRACT

To provide a solution by which adjustment of the depth display of the image can be easily carried out by the user at will in a technique for forming a 3-D image from plural images, from the plural feed images, one feed image is extracted as the reference feed image, with an object recognition process being carried out to extract the object region having the prescribed characteristic features. The reference feed image IL is displayed on the display unit  108  together with the markers MK indicating the object regions, and the user selects one object region. A region that is similar in image content with the selected region is detected from each other feed image, with the images being shifted so that the regions overlap each other.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No.2012-085693 filed on Apr. 4, 2012. The entire disclosure of JapanesePatent Application No. 2012-085693 is hereby incorporated herein byreference.

BACKGROUND

1. Technical Field

The present invention relates to an image processing apparatus and animage processing method that form a 3-D viewable synthetic image fromplural feed images. In particular, the invention relates to processingfor forming a 3-D viewable synthetic image by the combination with alenticular lens.

2. Background Technology

As a scheme for the 3-D display of images, the method that exploits theparallax of vision by two eyes has been adopted in practicalapplications. For example, for a lenticular image, from the pluralimages taken from views being different, rectangular images are cut out,with a parallax-attached synthetic image being formed as theserectangular images are arranged side by side sequentially correspondingto the configuration of the views. As the synthetic image is presentedvia a lenticular lens, there is a parallax between the images that reachthe left eye and the right eye, respectively, so that the object can beviewed in 3-D.

Schemes that automatically form the lenticular image or some other 3-Dviewable synthetic image from the plural feed images taken fromdifferent views have been proposed (for example, see Patent Document 1and Patent Document 2).

Japanese Laid-open Patent Publication No. 2004-104329 (for example, FIG.8) (Patent Document 1) and Japanese Laid-open Patent Publication No.2004-129186 (for example, FIG. 7) (Patent Document 2) are examples ofthe related art.

SUMMARY Problems to Be Solved by the Invention

When a 3-D viewable synthetic image is formed, by adjusting the relativepositions of the feed images for synthesis, it is possible to adjust thedepth positions of the persons, articles, etc., included in the image,that is, it is possible to adjust the degree of pop outs or recessesfrom the image plane. In particular, when the object is set on the sameplane as the image plane, it is possible to form a 3-D image that looksas the object is in focus. Here, if there is a function for the user toselect the object in focus at will, it will be convenient for the user.However, according to the well-known image processing technology, theconfiguration in the depth direction is uniquely fixated, and there isno margin for the user to make any selection, or the user has to carryout a manual operation to judge whether the relative position of theimage is finely adjusted or not. Anyway, such a function is not easilyavailable for the user.

Some embodiments of the invention provide solutions to the statedproblems by allowing adjustment of the depth display of the imagecarried out easily by the user at will.

Means Used to Solve the Above-Mentioned Problems

An embodiment of the invention provides an image processing apparatusthat forms a 3-D viewable synthetic image via a lenticular lens on thebasis of the plural feed images having parallax with respect to eachother, and has the following means: a means that receives the regionassignment input from the user who assigns a portion of the region in areference feed image among the feed images, a means that detects theregion corresponding to the region assigned in the reference feed imagefrom feed images other than the reference feed image, and a syntheticimage forming means that has the rectangular images cut out from theplural feed images, respectively, side by side and that forms thesynthetic image having the smallest parallax between the assigned regionand the corresponding region.

According to the invention, the “corresponding region” in other feedimages corresponding to the assigned region of the reference feed imagerefers to the region with the highest similarity to the image in theassigned region of the reference feed image. In other words, the regionwith points in the feed image that correspond most to the various pointsin the assigned region is the “corresponding region” of the feed image.

According to the invention with the aforementioned configuration, whenthe user selects any region in a feed image, the corresponding regioncorresponding to the assigned region is detected from the other feedimage and, at the same time, a synthetic image is formed so that theparallax between the assigned region and the corresponding region is ata minimum. As a result, when the synthetic image is viewed in 3-D, theobject in the assigned region becomes the most vivid, that is, theobject is displayed as a focused image. In this case, the user can onlycarry out the operation in assigning the region of the object that isout of focus, with the subsequent processing operation being carried outautomatically. Consequently, the invention provides an option for theuser, by which the user can easily adjust the depth display of the imageat will.

According to the invention, for example, the following scheme can alsobe adopted: among the reference feed image and the other feed image, thesynthetic image forming means cuts out the rectangular images from theoverlapped portion of the images where the assigned region of thereference feed image and the corresponding region of the other feedimage overlap each other. In addition, the following scheme can beadopted: the scheme has a shift quantity determining means thatdetermines the image shift quantity corresponding to the other feedimage with respect to the reference feed image needed to make thecorresponding region of the other feed image overlap with the assignedregion of the reference feed image, and the image forming means cuts outthe rectangular images from the overlapped image portion as the otherfeed image is shifted by an image shift quantity with respect to thereference feed image.

With any of these configurations, the synthetic image is formed bycutting out the rectangular images after adjusting the relative positionbetween the feed images so that the parallax at a the minimum for theassigned region of the reference feed image and the corresponding regionof the other feed image. Consequently, it is possible to quite vividlydisplay the object included in the region assigned by the user.

One can also adopt a scheme in which, from the regions included in theother feed image and having the same size as that of the assignedregion, the detecting means takes the region having the highestcorrelation with the assigned region as the corresponding region of theother image. In this way, by having a region with a high correlationwith the assigned region and with the same size as that of the assignedregion as the corresponding region, it is possible to form an imagewhere the desired object by the user can be in focus at a highprobability. As a specific means for carrying out this operation, forexample, the correlation between the regions can be determined by, forexample, the area correlation computing.

One can also adopt a scheme in which there is an image recognition meansthat extracts the regions having the prescribed characteristic featuresas the candidate regions by the image recognition process in thereference feed image, with the receiving means having a display unitthat displays the reference feed image in a form that allowsidentification of the candidate regions; at the same time, the userselects one region from the candidate regions as the assigned region. Asa result, the user simply selects from the candidate regions displayedon the display unit, and an image with the region in focus isautomatically obtained. Here, the well-known face recognition processingtechnology can be adopted as the image recognition treatment, and thehuman face can be taken as the candidate region that is the “regionhaving the prescribed characteristic features”. In this way, a syntheticimage having the prescribed person image included in the feed images infocus is obtained.

Also, the image processing apparatus of the invention can have aprinting means that prints out the synthetic image. By combining theprinted image with the lenticular lens, it is possible to provide theuser with a printed copy of the 3-D image having the desired depthappearance.

As another embodiment, the invention provides an image processing methodthat forms a 3-D viewable synthetic image via a lenticular lens on thebasis of the plural feed images having parallax with respect to eachother. This image processing method has the following steps ofoperation: an assignment receiving step in which a portion of the regionin the reference feed image as one of the feed images is assigned by theuser, a detection step in which the corresponding region correspondingto the assigned region is detected from a feed image other than thereference feed image, and an image forming step in which the syntheticimage is formed by setting side by side the rectangular images cut outfrom the overlapped portion where the assigned region of the referencefeed image and the corresponding region of the other feed image areoverlapped with each other.

Just as the invention related to this image processing apparatus,according to the invention of the image processing method with theconfiguration, it is possible to provide a synthetic image that can mostvividly display the object included in the region assigned by the userand the user only needs to assign the region to obtain such a syntheticimage.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the attached drawings which form a part of thisoriginal disclosure:

FIG. 1 is a diagram illustrating the printing system that adopts anembodiment of the image processing apparatus related to the invention;

FIG. 2 is a flow chart illustrating the 3-D image printing mode in thisembodiment;

FIG. 3 is a diagram illustrating an example of the feed images;

FIG. 4 is a diagram illustrating an example of the displayed image;

FIG. 5 is a diagram illustrating the principle of the process forsearching the corresponding region;

FIG. 6 is a flow chart illustrating an example of the specific operationof the searching process of the corresponding region;

FIGS. 7A and 7B are diagrams illustrating the concept of image shift andtrimming; and

FIGS. 8A and 8B are diagrams illustrating another embodiment of the 3-Dimage printing mode.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 is a diagram illustrating the printing system adopting anembodiment of the image processing apparatus of the invention. Accordingto this printing system, the image data acquired by picture taking usinga digital camera 200 are sent by a memory card M, a USB (universalserial bus) cable, a wireless LAN (Local Area Network), or the like to aprinting apparatus 100, and are printed out by the printing apparatus100. That is, here, the user uses the digital camera 200 to takepictures to generate the image data, and the image data are read andprinted as is on the printing apparatus 100 in the so-called directprinting mode. However, the invention is not limited to this printingsystem. That is, the invention can also be adopted in a printing systemin which the image data generated by the digital camera 200 are fetchedinto a personal computer, a cell phone or the like, and the image dataare then sent from the personal computer to the printing apparatus 100for printing. However, the invention is not limited to the system havingboth the digital camera 200 and the printing apparatus 100. Theinvention can also be adopted in any image processing apparatus for theimage data in a general sense.

As shown in the drawing, the digital camera 200 has a CPU (centralprocessing unit) 201, a ROM (read-only memory) 202, a RAM (random accessmemory) 203, CCDs (Charge Coupling Devices) 204L, 204R, a graphicprocessor (GP) 205 and an interface (I/F) 206 connected with each othervia a bus 207. Information can be exchanged among them. Corresponding tothe program stored in the ROM 202, the CPU 201 executes various types ofarithmetic and logic operations, while it controls the digital camera200. In this case, the data that are temporarily needed are stored inthe RAM 203.

Also, the CCDs 204L, 204R convert the optical images from the objectwith light collected by the optical systems 208L, 208R into electricsignals for output. More specifically, the optical image collected bythe optical system 208L is incident on the CCD 204L, while the opticalimage collected by the optical system 208R is incident on the CCD 204R.The optical systems 208L, 208R are separated from each other on theleft/right portions of the case of the digital camera 200, respectively.More specifically, the optical system 208L is arranged on the left sidewith respect to the object on the front surface of the case of thedigital camera 200, while the optical system 208R is arranged on theright side with respect to the object. Consequently, parallax existsbetween the images taken by the CCDs 204L, 204R, respectively.

The optical systems 208L, 208R each are made of plural lenses andactuators. The actuators work to adjust the focus or the like while theoptical images of the object are formed by the plural lenses on thelight receiving surfaces of the CCDs 204L, 204R, respectively.

The digital camera 200 can selectively executes the following modes: a3-D image pickup mode, in which the two CCDs 204L, 204R are used to takea pair of pictures with a parallax between them, and the well-knownimage pickup mode in which any of the CCDs is used to carry out imagepickup. The one pair of image data taken in the 3-D image pickup modeare stored as a correlated pair. In the process for forming the 3-Dviewable synthetic image, to be explained later, the image taken by theCCD 204L is used as the feed image for the left eye and the image takenby the CCD 204R is used as the feed image for the right eye.

In addition, the GP 205 executes the image processing for display on thebasis of the display command sent from the CPU 201, with the obtainedimage data for display being sent to the liquid crystal display (LCD)209 for display.

The I/F 206 provides the input/output function of the digital camera200. When the information is exchanged between the operation button 210,the gyro sensor 211, and the I/F circuit 212, it performs appropriateconversion for the format of the data for display. The operation button210 connected to the I/F 206 has the buttons for power supply, modeswitching, shutter, etc., as well as the input means that can set thevarious types of functions. As a result, the user can control thedigital camera 200 at will for the desired operation. Here, the gyrosensor 211 generates and outputs a signal indicating the angle (theangle with respect to the horizontal plane) of the camera main body whenthe image of the object is taken by the digital camera 200. The digitalcamera 200 generates the various types of information (such as exposure,information about the object, etc.) in the image pickup operationincluding the angle of the camera main body.

According to the present embodiment, the digital camera 200 has astructure that allows a description of the image pickup information asthe Exif (Exchangeable Image File Format) information and the generationof the image file attached on the image data. The structure of the Exifimage file basically is the well-known JPEG (Joint Photographic ExpertGroup) image format. In this image file, the thumbnail image, the imagepickup related data, and other data are buried in the format accordingto the JPEG code. In addition, it has the function of forming andrecording the image file (MPO file) on the basis of the MP (MultiPicture) that has plural still picture image data recorded in one imagefile.

The I/F circuit 212 is an interface for reading the information with thememory card M inserted in the card slot 213. In addition, the I/F 206also has the function of connecting with the USB, wireless LAN, andother external equipment not shown in the drawing, and it allowsexchange of the image file with the printing apparatus 100 either withwires or wirelessly.

The printing apparatus 100 is an apparatus that prints out the imagestaken by the digital camera 200. The printing apparatus has thefollowing configuration. In the printing apparatus 100, the CPU 101, theROM 102, the RAM 103, the EEPROM (electrically-erasable-programmableROM) 104, the GP 105, and the I/F 106 are connected with each other viathe bus 107. The information can be exchanged between them. The CPU 101executes the various types of arithmetic and logic operationscorresponding to the programs stored in the ROM 102 and the EEPROM 104,and at the same time, it controls the various sections of the printingapparatus 100. In addition, the CPU 101 has the program and the data asthe subject for execution temporarily stored in the RAM 103; it also hasthe data to be maintained even after turning off the power supply of theprinting apparatus stored in the EEPROM 104. Besides, as needed, the CPU101 sends the display command to the GP 105, the GP 105 executes theimage processing for display corresponding to the display command, andthe result of the process is sent to the display unit 108 for display.

The I/F 106 is an apparatus that performs appropriate conversion of thedata display format when information exchange is carried out between theoperation button 109, the card I/F circuit 110, and the printer enginecontroller 111. In the printing apparatus 100, the operation button 109has a configuration for pressing to make a menu selection, etc., of theprinting apparatus 100. Also, the card I/F circuit 110 is connected tothe card slot 112 and the image file generated by the digital camera 200is read from the memory card M inserted into the card slot 112. Also,the I/F 106 has the function of connection with the USB, the wirelessLAN, and other external equipment not shown in the drawing, with theimage file being exchanged with the digital camera 200 either by wiresor wirelessly.

The display unit 108 has a touch panel arranged on the surface of thedisplay unit made of, for example, an, LCD. In addition to displayingthe image data sent from the GP 105 on the display unit, the operationinput data input by the user on the touch panel are output to the I/F106.

For the printing apparatus 100, as the image data are received via thememory card M or by data communication, various processes are carriedout by the CPU 101 and, at the same time, the printer engine controller111 controls the printer engine 113 and the image corresponding to theimage data is printed. In the following, the 3-D image printing modewill be explained. According to this mode, a 3-D viewable syntheticimage is formed from the image data corresponding to one pair ofleft/right feed images taken by the digital camera 200 in the 3-D imagepickup mode and, in combination with a lenticular lens, the image isprinted on a recording sheet to form a lenticular image.

In addition, it is possible to execute the various printing operationsthat can be carried out by the printers of this type. However, theseprinting operations are well known technologies and they can be adoptedin the present embodiment, so they will not be explained in thisspecification. In addition, the principle of the 3-D viewing by thelenticular image and the method of the principle for forming an imagefrom plural feed images is also well known. Consequently, they will notbe explained in detail here.

FIG. 2 is a flow chart illustrating the 3-D image printing mode in thisembodiment. FIG. 3 is a diagram illustrating an example of the feedimages. In this printing mode, first of all, the feed images as theorigin of the 3-D image are acquired (step S101). The feed images shouldbe plural images having parallax with respect to each other. Forexample, it is possible to use one pair of images taken by the digitalcamera 200 in the 3-D image pickup mode. However, the feed images arenot limited to this type. For example, for a group of plural imagestaken for the same object from different views, such as a group ofimages formed using, for example, the computer graphics technology, thetechnology to be described below can also be adopted. The number ofimages that include one group of feed images can be 2 or larger.

Here, an explanation will be made on the case in which two images aretaken by the digital camera 200 in the 3-D image pickup mode. As shownin FIG. 3, in the 3-D image pickup mode, as shown in FIG. 3, two imagesIL, IR are taken for the same object from different views. Here, theimage IL is an image taken by the CCD 204L arranged on the left side inthe digital camera 200; the image is used as the feed image for the lefteye when a lenticular image is formed. On the other hand, the image IRis the image taken by the CCD 204R arranged on the right side in the CCD204L; the image is used as the feed image for the right eye when alenticular image is formed.

The main objects shared in these images include two persons on theleft/right sides, a yacht at the center, and a hill on the upper-leftcorner. Between the feed image IL for the left eye and the feed image IRfor the right eye, there is a delicate difference in the positions ofthe objects corresponding to the distance between the camera and theobjects in the image pickup operation. That is, for a, more distantobject, there is little difference in the position between theleft/right feed images IL, IR. Thus, the nearer the object is to thecamera, the larger the difference in the position of the objectcorresponding to the subject for picture taking.

In the example shown in FIG. 3, the person on the right side is on thefront, and its difference D1 in position between the left/right feedimages in the lateral direction is the largest. This difference thendecreases in the order of the difference D2 for the position of theleft-side person on the back side, followed by the difference D3 of theposition of the yacht further on the back side. On the other hand, for ahill that is far away, there is little difference in the position. Inaddition, for the actual images, the difference in position caused bytilting of the camera, etc., can be added. When plural feed images aretaken by individual cameras, and when picture taking is carried out by asingle-lens camera via a 3-D adopter, the differences caused by suchposition shifts can also be added.

When the feed images formed are directly used without trimming to formthe 3-D viewable synthetic image, just as in the image pickup state, thefarthest object (the hill) has zero parallax, while as the object movesto the front, the parallax increases, with the front object having thelargest parallax, appearing to pop out towards the viewer. On the otherhand, from the viewpoint of vividness of the objects, there is atendency that the far-away object free of parallax appears to be themost vivid, while as the object moves to the front, the image becomesblurred. For this type of 3-D image, while the object free of parallaxis positioned on the same plane as the image plane and appears vivid,the object with parallax is positioned ahead (or behind) the image planeto display a depth sensation, while as the back-and-forth range that canbe displayed is limited, the depth in the image pickup mode cannot bereproduced with a high fidelity, so that such an object is not vivid.

As to be explained below, according to the present embodiment, the usercan select the object to be vividly displayed. Consequently, using thesame effort as that of focusing by applying the auto focus function inimage pickup operation, the user simply selects and inputs the object tobe vividly displayed, and, without any fine adjustment of the image bythe user operation, it is possible to obtain a 3-D image with thedesired object in focus.

The operation for this function can be explained with reference to FIG.2. Here, any one of the plural feed images taken as is selected and, forthe selected feed image, the recognition process is carried out for theobject included in the feed image (step S102). Here, suppose the feedimage IL for the left eye is selected. The object recognition processcan be carried out using the well known technology, such as the facerecognition process in which the region having the characteristicfeatures of the human face is extracted, or the process in which eachpixel has a pixel value converted to binary values with a prescribedthreshold, with the closed region being extracted. Then, together withthe information indicating the region of the object extracted from thefeed image IL (such as the frame indicating the contour of the objectregion), the feed image IL is shown on the display unit 108 (step S103).

FIG. 4 is a diagram illustrating an example of the displayed image.Here, on the display unit 108, the feed image IL that has been selectedis displayed. Here, hook shaped markers MK indicating the four cornersof the regarding shaped region recognized as the object are overlappedand displayed. In this example, the faces of the left/right persons andthe yacht in the central upper portion are recognized as the objects anddisplayed.

Here, the user selects the object to be displayed most vividly fromamong the recognized objects by touching the object displayed on thedisplay unit 108 with its surface made of a touch panel. As a result,below the image, a message prompting selection of the object isdisplayed for the user.

In this state, the message remains for the user to make the operationinput for selecting the object (step S104). As the user touches thetouch panel to select a certain object as the effective one, the regionof the selected object is taken as the assigned region of the feed imageIL for the left eye and, in the remaining one feed image, that is, thefeed image IR for the right eye, the corresponding region having animage content similar to that of the assigned region is searched (stepS105). Here, as shown in FIG. 4, the face of the person on the rightside is selected by the user.

FIG. 5 is a diagram illustrating the principle of the process forsearching the corresponding region. FIG. 6 is a flow chart illustratinga specialist operation example. Here, the assigned region R1 selected inthe feed image IL for the left eye is a rectangular region having Pxpixels in the lateral direction and Py pixels in the vertical direction.In order to detect the corresponding region in the feed image IR for theright eye corresponding to this, the upper-left corner of the feed imageIR for the right eye is taken as the origin (0, 0) and, on thecoordinate plane with the X-axis in the lateral direction and Y-axis inthe vertical direction, a window W2 having the same size as that of theassigned region R1 is set. That is, the window W2 has Px pixels in thelateral direction (X-direction) and Py pixels in the vertical direction(Y-direction).

Next, the similarity is determined between the image content in thewindow W2 set on the feed image IR for the right eye and the imagecontent of the assigned region R1 assigned on the feed image IL for theleft eye. The similarity can be quantitatively evaluated by the heightof the correlation between the two image contents. For example, thesimilarity can be determined by the well known area correlationcomputing.

The position of the window W2 can be set with various options inside thefeed image IR for the right eye. Among them, the region in the feedimage IR for the right eye specified by the window W2 with the highestsimilarity between the image content in the window W2 and the imagecontent in the assigned region R1 can be taken as the regioncorresponding to the assigned region R1. The corresponding region can bespecified by the information obtained by specifying the coordinates ofthe four corners of the window W2 in this case. For example, this regioncan be represented by the coordinate position at one of the four cornersand the center of gravity and the window size, or by the coordinates ofthe two apexes located on the diagonal line of the rectangular shape.

In the following, a specific process example will be explained withreference to FIG. 6. At first, the coordinate parameters x, y thatassign the position of the window are set at their initial values (0)(steps S201, S202). The coordinates (x, y) specified by these coordinateparameters are taken as the position of the apex at the upper-leftcorner, and the window W2 having the same size as that of the assignedregion R1 (with Px pixels in the lateral direction and Py pixels in thevertical direction) is set inside the feed image IR for the right eye(step S203).

The area correlation between the various pixels in the window W2 set asand the pixels in the assigned region R1 is carried out to determine thesimilarity of the image contents (step S204). More specifically, thefollowing operation can be carried out. For all of the pixels in theregion, computing is carried out to determine the difference between thepixel value of a pixel selected from within the assigned region R1 andthe pixel value of the pixel at the position corresponding to it in thewindow W2, with the integrated value of the absolute value of thedifference then being determined. The smaller the integrated value, thehigher the correlation of the image contents. That is, suppose the imagecontents are entirely in agreement with each other, the difference inthe pixel value between the corresponding pixels becomes zero, and theintegrated value also is zero. The lower the similarity between theimage contents, the larger the difference between the pixels, and theintegrated value of the absolute value also increases. With such anarithmetic and logic operation, it is possible to determine thesimilarity between the image contents.

The determined integrated value is then compared with the stored minimumvalue (step S205). As shown in FIG. 6, in the searching treatment, aloop process is carried out and the integrated value is repeatedlycomputed repeatedly. The minimum value of the integrated values computedin the various loops and the coordinate position (x, y) of the windowset when the minimum value is computed are stored. If the newly computedintegrated value is smaller than the minimum value (YES in step S205),the stored minimum value is refreshed to the integrated value and, atthe same time, the coordinate position (x, y) of the window in this caseis refreshed and stored (step S206). On the other hand, if theintegrated value computed in the new loop is larger than the storedminimum value (NO in step S205), the minimum value and the windowcoordinate position are not refreshed.

The processes carried out in steps S202 through S206 are executed ineach step by incrementing the coordinate parameter x by 1 until theright end of the feed image IR of the window is reached (step S207)(step S211); at the same time, the processes carried out in steps S202through S207 are executed in each step by incrementing the coordinateparameter y by 1 until the lower end of the feed image IR of the windowis reached (step S208) (step S212). In this way, the region with animage content having the highest similarity with the assigned region R1is searched out from the entirety of the feed image IR for the righteye.

That is, the region in the window specified by the coordinate positionof the window and the window size stored at the end time point of theloops is detected as the “corresponding region” in the feed image IR forthe right eye having an image content corresponding to the image contentof the assigned region R1. In the following, the corresponding region inthe feed image IR for the right eye corresponding to the assigned regionR1 assigned by the feed image IL for the left eye will be represented bykey R2.

Now, return to FIG. 2. The image shift quantity of the feed image IR forthe right eye with respect to the feed image IL for the left eye neededto have a perfect overlap between the assigned region R1 and thecorresponding region R2 determined as is then computed (step S106).Next, from the region that repeats when the feed image IR for the righteye is shifted by the image shift quantity determined as and overlappedwith the feed image IL for the left eye, the region corresponding to thesize of the string unit to be formed is extracted by trimming (stepS107).

FIGS. 7A and 7B are diagrams illustrating the concepts of the imageshift and trimming. In the example shown in FIG. 7A, when the feed imageIR for the right eye is shifted by ΔX pixels in the (−X) direction andby (−ΔY) pixels in the (−Y) direction with respect to the feed image ILfor the left eye, the assigned region R1 of the feed image IL for theleft eye and the corresponding region R2 of the feed image IR for theright eye are perfectly overlapped with each other. That is, the valueof the image shift quantity to be determined is ΔX in the X-axisdirection and ΔY in the Y-axis direction, and, as they are attached withsigns, they can represent the image shift quantities in the twodirections (positive/negative directions). The image shift quantity canbe determined from the coordinate position of the assigned region R1 inthe feed image IL for the left eye and the coordinate position of thecorresponding region R2 for the feed image IR for the right eye.

For example, suppose the upper-left corner of the assigned region R1 inthe feed image IL for the left eye has the coordinates of (x1, y1), andthe upper-left corner of the corresponding region R2 in the feed imageIR for the right eye has the coordinates of (x2, y2), the image shiftquantity Sx in the X-direction and the image shift quantity Sy in theY-direction of the feed image IR for the right eye with respect to thefeed image IL for the left eye can be represented as follows:

Sx=x1−x2

Sy=y1−y2

Here, if the image shift quantity has a positive value, the image isshifted in the direction of the coordinate axis; if the image shiftquantity has a negative value, the image is shifted in the negativedirection of the coordinate axis, so that the assigned region R1 of thefeed image IL for the left eye and the corresponding region R2 of thefeed image IR for the right eye are overlap each other.

The image in the repeated region when the feed image IR for the righteye is shifted to overlap with the feed image IL for the left eye can beused in forming the 3-D image. From this overlapped region, the regioncorresponding to the aspect ratio of the image to be printed isextracted; this region can be set as the effective printing region RPadopted in the synthetic image.

Now return to FIG. 2. From the images of the trimmed effective printingregions RP among the feed image IL for the left eye and the feed imageIR for the right eye, the rectangular images as the images for the lefteye and the images for the right eye are cut out and are alternatelyarranged to form a synthetic image (step S108). The technology forforming the image appropriate for the 3-D viewing by synthesis from therectangular images cut out from the plural feed images after trimming isa well known technology, so that it will not be explained again here.

The synthetic image prepared as is shown on the display unit 108 (stepS109), and the user is asked to check the image content. If the userexpresses OK by manipulating the input (step S110), the printer engine113 is turned on, the synthetic image is printed on the lenticular sheet(step S111), and the process comes to an end. On the other hand, if theuser does not express satisfaction (NO in step S110) for the displayedimage content, the process returns to step S103, and the user once againselects the object.

In order to provide a 3-D synthetic image (lenticular image), one canalso adopt a scheme in which a lenticular lens is bonded on a flatrecording material after printing of the synthetic image, or thesynthetic image is printed on the other side (a flat surface) of a sheetshaped recording material (“lenticular sheet” in this specification)having a lenticular lens arranged on one side. In the latter case, ofcourse, the longitudinal direction of the rectangular images should bein agreement with the longitudinal direction of the lenticular lens. Inaddition, as the printed surface is viewed through the lenticular lensfrom the back side of the printed surface for displaying the image in3-D, the printed surface should be transparent, and the printing data ina state flipped from the synthetic image shown on the display unit 108should be sent to the printer engine 113.

As explained above, according to this embodiment, when a 3-D viewablesynthetic image is formed from plural feed images having differentviewpoints, the user selects and assigns the region of the object to bedisplayed most vividly from one feed image. Next, the correspondingregion having an image content similar to that of the assigned region onthe feed image is detected from the other feed image, and these regionsare overlap each other. The portion overlapped with the feed image istrimmed to form a synthetic image. As a result, when the synthetic imageis viewed in 3-D, the assigned region in one feed image and thecorresponding region on the other feed image have the minimum parallaxbetween them. Consequently, the image object in the assigned region canbe positioned on the same surface as the image surface and can bedisplayed most vividly, while the other image objects are displayed tothe front side or back side corresponding to the parallax values, sothat they are displayed with a depth appearance.

For example, as shown in FIG. 7A, when the face region of the person onthe right side is designated as the assigned region R1, image shift andoverlap lead result in the face region of the person on the right sidebecoming more vivid, while the objects corresponding to the othersubjects at other distances (the person on the left side, the yacht andthe far-away hill, etc.) display significant position offsets.Consequently, for the synthetic image obtained in this state, while theportion near the face of the person on the right side is positioned onthe same plane as the image plane and can be displayed vividly, theother objects are displayed a little blurred on the front side or backside, respectively, from the face of the person on the right side.

Also, for example, as shown in FIG. 7B, when the face region R3 of theperson on the left side is designated as the assigned region, for thesynthetic image, the person on the left side can be displayed vividly,while for the person on the right side, the position offset quantitybetween the image for the left eye and the image for the right eyebecomes larger, and the person's image becomes blurred. In this way, theuser can adjust where to display the image in the depth direction of theimage according to his/her preference.

The operation that the user needs to carry out to obtain such an imageis simply the designation of the assigned region in one feed image, andthere is no need to perform the manual operation to adjust the imageshift quantity. Consequently, according to this embodiment, the user canadjust the image easily at will to obtain the desired 3-D image. This istrue even when there are plural feed images. In particular, the regionrecognized by the object recognition process for the feed image isdisplayed together with the feed image, and the user can use the touchpanel to select them. Consequently, the selection operation can becompleted simply as the user selects the favored register on thedisplay, so that the operation becomes very simple.

In addition, the process for detecting the region corresponding to theassigned region on a feed image from the other feed image can be carriedout by determining the difference in the pixel value between the regionfetched from the other feed image and the assigned region, followed byintegration, then the region with the minimum integrated value issearched. This is a very simple arithmetic and logic operation, so thatthe process contents can be well handled by a processor without a highprocessability. Consequently, the use of this technology does not boostthe cost of the apparatus, and the functions can be realized even inlow-price products.

In the embodiment, the image shift quantity is set so that the assignedregion corresponding to the object assigned by the user on one feedimage and the corresponding region on the other feed image correspondingto this assigned region are in agreement with each other on thesynthetic image. That is, the synthetic image is formed with theassigned object vividly positioned on the image plane. However, thedepth display method for a lenticular image can also be carried out asfollows in addition to this scheme.

FIGS. 8A and 8B are diagrams illustrating another embodiment of the 3-Dimage printing mode. FIG. 8A shows the relationship between the positionoffset quantity in the left/right direction of the corresponding objectbetween the image for the left eye and the image for the right eye andthe pop out quantity of the object as it appears to pop out or recessfrom the image plane. For the abscissa, the (+) direction corresponds tothe case in which the corresponding object on the image for the righteye is located on the right side on the synthetic image with respect tothe object on the image for the left eye, and the (−) directioncorresponds to the opposite case. On the other hand, for the ordinate,the (+) direction corresponds to the state in which the object flies outforward from the image plane when viewed as 3-D, and the (−) directioncorresponds to the state in which it recesses back from the image plane.As shown in FIG. 8A, the pop out quantity of the corresponding objectfrom the image plane varies corresponding to the left/right offsetquantity of the corresponding object between the image for the right eyeand the image for the left eye.

Here, if the left/right offset quantity of the object is too large, theviewer sees the images of the object incident into the left/right twoeyes as double views instead of recognition as the images of the sameobject. That is, there is a limit for the range of distance in the depthdirection where the images of the object incident to the two eyes can beviewed as a single object without blurring by the viewer (the imagefusible range). In the 3-D image display using a lenticular sheet, theimage fusible range is limited by the characteristics of the lenticularlens and the printing precision, etc.

In other words, when the left/right offset quantity between the imagefor the right eye and the image for the left eye of the object assignedby the user is within the image fusible range, it is possible to makevarious types of 3-D displays without sacrificing the vividness of thecorresponding object. For example, when the left/right offset quantityQ1 corresponding to the lower limit of the image fusible range is givento the corresponding object, the corresponding object is positioned atthe deepest site within the image fusible range. On the contrary, whenthe left/right offset quantity Q2 corresponding to the upper limit ofthe image fusible range is given to the object, the corresponding objectis positioned on the front side in the image fusible range. If theleft/right offset quantity is larger than that, the image of the objectbecomes blurred. However, by setting the value of the left/right offsetbetween Q1 and Q2 appropriately, it is possible to exploit the depththat allows the display by the lenticular sheet to the upper limit. Inparticular, for the image including plural subjects for image pickupwith various different locations in the depth direction, it is possibleto efficiently display their back-and-forth relationship.

As the user sets the pop out quantity of the image, it is possible toprovide a lenticular image with the depth display corresponding to thepreference of the user. Because a simple operation by the user canrealize such a function, the following schemes can be adopted. As thefirst example, the options that can be selected by the user are limited,and the depth position of the assigned object is selected from thefront, the back, and the same plane as the image plane in the imagefusible range. As the second example, on the display screen for the userto assign the object (FIG. 4), a slider is displayed as a GUI (graphicaluser interface) part corresponding to the image fusible range. Next, asthe user manipulates the knob of the slider, it is possible tocontinuously set the left/right offset quantity within the image fusiblerange (Q1˜Q2). Of course, the operation schemes are not limited tothese.

The process that allows printing of the image with the depth display asselected by the user can be carried out by making partial changes forthe operation of the 3-D image printing mode shown in FIG. 2. Here,instead of step S106, the step S106 a shown in FIG. 8B is set. In stepS106 a, the left/right offset quantity between the image for the righteye and the image for the left eye for realizing the image pop outquantity set by the user is determined from the relationship shown inFIG. 8A, and the image shift quantity is computed so that the positionoffset quantity between the assigned region R1 and the correspondingregion R2 on the synthetic image is in agreement with the determinedvalue. On the basis of the image shift quantity obtained in this way,formation of the synthetic image and printing of the synthetic image arecarried out in the same way as in this embodiment, so that it ispossible to form a lenticular image with the depth display correspondingto the preference of the user.

In this way, the pop out quantity of the object corresponding to theregion assigned by the user from the image plane can be set by the userwithin the image fusible range corresponding to the characteristicfeatures of the printing apparatus and the recording media. Bydetermining the image shift quantity for realizing the left/right shiftquantity corresponding to setting and forming the synthetic image, it ispossible to provide the user with a lenticular image corresponding tothe preference of the user. This embodiment corresponds to the case inwhich the pop out quantity of the assigned object is fixed at zero.

As explained above, according to this embodiment, the printing apparatus100 works as the “image processing apparatus” of the invention, thedisplay unit 108 has the functions of the “display unit” and “receivingmeans” of the invention, and the printer engine 113 works as the“printing means” of the invention. Also, as the CPU 101 executes theprescribed control programs, the various functions of the “detectingmeans”, “shift quantity computing means”, “synthetic image formingmeans” and “image recognition means” of the invention are realized.

In the aforementioned embodiment, the feed image IL for the left eyecorresponds to the “reference feed image” of the invention, and the feedimage IR for the right eye corresponds to the “other feed image” of theinvention. They both correspond to the “feed images” of the invention.In the example of an image shown in FIG. 4, the various image regions(faces of the left/right persons and the yacht) specified by the markersMK correspond to the “candidate regions” of the invention.

However, the invention is not limited to this embodiment. As long as itsgist is observed, various modifications can be made. For example, inthis embodiment, the left/right feed images IL, IR taken from left/righttwo views are adopted to form the lenticular image. However, there is nospecific restriction on the number of feed images as long as it is 2 ormore. This technology can also be adopted in forming a synthetic imagefrom the feed images from plural viewpoints, respectively. Morespecifically, when one of the feed images is taken as a reference indesignating the assigned region on the feed image, the correspondingregions are detected from the other images, respectively, and the imageshift quantity with respect to the reference feed image is computed foreach feed image, with a synthetic image being formed on the basis of thetrimmed images from the overlapped portions of all of the feed images.

In this case, in principle, any of the plural feed images can be takenas the reference. However, from the viewpoint of practical applications,it is preferred that the feed image with a view near the center amongthe various views be taken as the reference. When the feed image with anouter view is the assigned region, the offset between the image contentwith the feed image from the view on the opposite side becomes larger,and the detection precision can be degraded when the correspondingregions corresponding to the assigned region are detected from the otherfeed images.

In this embodiment, the user selects the assigned region from among theobject regions recognized by the object recognition treatment. However,instead of this scheme, or accompanying this scheme, one can also adopta scheme in which the two apexes on the diagonal line are assigned, sothat any region can be taken as the assigned region. As far as themethod for presenting the recognized object region is concerned, it isnot limited to the scheme in which the four corners are indicated by themarkers MK. For example, one can also adopt a scheme in which thecorresponding region is surrounded by a regarding frame, and theluminance of the corresponding region is changed, or an other scheme.Also, the method for receiving the operation input is not limited to thetouch panel. One can also use the input from a keyboard or a mouse, orany other appropriate schemes.

In the embodiment, the image processing method of the invention isexecuted on a printing apparatus 100 that includes a printing systemtogether with a digital camera 200. However, the subjects of applicationof the invention are not limited to this. For example, the same imageprocessing method can also be adopted on a digital camera or printer, aportable terminal device, a personal computer, etc.

In this embodiment, the formed synthetic image is displayed on thedisplay unit 108 and, after the user is asked to check the imagecontent, printing is carried out on a recording sheet. However, such adisplay and checkup operation of the image are not a necessity.

The image processing apparatus and the image processing method of theinvention are preferably adopted in forming a 3-D viewable syntheticimage via, e.g., a lenticular lens. In addition to the case of printingand output of the image on a recording sheet, this apparatus and methodcan also be adopted when the image is displayed on a screen andpresented together with a lenticular lens.

What is claimed is:
 1. An image processing apparatus that forms a 3-Dviewable synthetic image via a lenticular lens based on plural feedimages having parallax from each other, comprising: a receiving unitthat receives a region assignment input from a user who assigns aportion of region in a reference feed image among the feed images; adetecting unit that detects a corresponding region corresponding to anassigned region assigned in the reference feed image from each otherfeed image other than the reference feed image; and a synthetic imageforming unit that has rectangular shaped images cut out from the pluralfeed images, respectively, side by side and forms the synthetic imagewith a smallest parallax between the assigned region and thecorresponding region.
 2. The image processing apparatus according toclaim 1, wherein among the reference feed image and the other feedimage, the synthetic image forming unit cuts out the rectangular shapedimages from an overlapped portion of those images where the assignedregion of the reference feed image and a corresponding region of theother feed image are overlapped with each other.
 3. The image processingapparatus according to claim 2, further comprising a shift quantitydetermining unit that determines an image shift quantity correspondingto the other feed image with respect to the reference feed image neededfor having the corresponding region of the other feed image to overlapwith the assigned region of the reference feed image, wherein the imageforming unit cuts out the rectangular shaped images from an imageoverlapped portion overlapped as the other feed image is shifted by animage shift quantity with respect to the reference feed image.
 4. Theimage processing apparatus according to any one of claims 1 through 3,wherein from the regions included in the other feed image and having asame size as that of the assigned region, the detecting unit takes theregion with a highest correlation with the assigned region as thecorresponding region of the other image.
 5. The image processingapparatus according to claim 4, wherein the detecting unit determinesthe correlation between the region in the other feed image and theassigned region by area correlation computing.
 6. The image processingapparatus according to any one of claims 1 through 5, further comprisingan image recognition unit that extracts regions having a prescribedcharacteristic features as candidate regions by image recognitiontreatment in the reference feed image, wherein the receiving unit has adisplay unit that displays the reference feed image in a form thatallows identification of the candidate regions, and, at a same time, theuser selects one region from the candidate regions as the assignedregion.
 7. The image processing apparatus according to any one of claims1 through 6, further comprising a printing unit that prints out thesynthetic image.
 8. An image processing method that forms a 3-D viewablesynthetic image via a lenticular lens based on plural feed images havingparallax from each other, comprising: assigning a portion of region in areference feed image as one of the feed images, detecting acorresponding region corresponding to an assigned region from each otherfeed image other than the reference feed image, forming a syntheticimage by setting side by side rectangular shaped images cut out from anoverlapped portion where the assigned region of the reference feed imageand a corresponding region of the other feed image are overlapped witheach other. detecting the region corresponding to the assigned regionfrom a feed image other than the reference feed image, and forming thesynthetic image by setting side by side the rectangular images cut outfrom the overlapped portion where the assigned region of the referencefeed image and the corresponding region of the other feed image areoverlap each other.